Systems Group , Department of Computer Science , ETH Zurich Efficient Scan in Log - Structured Memory Data Stores

Novel storage solutions like RAMCloud promise never seen before latency by storing all their data in main memory. To efficiently store and retrieve objects from memory some systems reuse ideas from log-structured file systems and utilise a log like data structure to hold the data objects in DRAM. Instead of updating data in place, with log-structured memory any modification to the data generates a new data entry which is appended sequentially to the log’s head. An indexing structure (often a hash table) mapping an object’s key to the location in the log of its most recently written value is used to enable fast random read access to any object by key. Due to the append-only nature the log grows over time and as such a garbage collection algorithm is needed to reclaim space by stripping inaccessible objects from the log. While those systems provide exceptional random access performance (retrieving or writing an object with arbitrary key) they lack an efficient way to perform a scan over the whole data set. Scanning over all currently stored objects usually requires traversing the hash table and retrieving its associated data from the log. This approach has suboptimal performance due to poor CPU cache locality as the data tuples are scattered throughout the log under normal load. Main goal of this master thesis is to devise an optimized log-structured memory data store offering a dedicated scan operation. This scan operation should be as efficient as possible while trying to preserve the very high random access performance the log-structured data layout enables in the first place. The system consists of the implementation of two different approaches: A baseline approach modeling the current state of the art log-structured memory algorithms and one novel approach optimized for both scan and random access operations. These implementations build on top of a full featured data store to read and write key-value pairs in DRAM including optimized memory allocation and garbage collection algorithms. Finally, the different approaches are evaluated regarding their performance under different workloads.